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The Effects of the Louisiana Scholarship Program on 
Student Achievement after Two Years 


Abstract 

The Louisiana Scholarship Program (LSP) is a statewide initiative offering publicly-funded 
vouchers to enroll in local private schools to students in low-performing schools with family 
income no greater than 250 percent of the poverty line. Initially established in 2008 as a pilot 
program in New Orleans, the LSP was expanded statewide in 2012. This paper examines the 
experimental effects of using an LSP scholarship to enroll in a private school on student 
achievement in the first two years following the program’s expansion. Our results indicate that 
the use of an LSP scholarship has negatively impacted both ELA and math achievement, 
although only the latter estimates are statistically significant. Moreover, we observe less negative 
effect estimates in the second year of the program. 

Keywords: school vouchers, student achievement, randomized control trial, experiment, school 
choice 
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The Effects of the Louisiana Scholarship Program on 
Student Achievement after Two Years 


1. Introduction 

The Louisiana Scholarship Program (LSP) is a statewide school voucher initiative that provides 
public funds for low-income students in underperforming public schools to attend participating 
private schools. 1 Originally piloted in New Orleans in 2008, the statewide expansion of the LSP 
in 2012-13 allowed almost 5,000 low- to moderate-income students across the state of Louisiana 
to transfer out of their traditional public schools and into private schools at state expense. 2 The 
empirical evidence presented here examines how the LSP has impacted student achievement two 
years after the statewide expansion. 

Our analysis uses the results of the oversubscription lotteries for nearly 10,000 eligible 
applicants to analyze the achievement impacts of LSP as a randomized control trial (RCT). In 
particular, we use admission lotteries as instrumental variables to estimate the effect of using an 
LSP scholarship to enroll in a private school for applicants to oversubscribed lotteries who were 
induced to attend a private school as a result of winning the lottery. Our analysis uses student- 
level data obtained via a data-sharing agreement with the state of Louisiana. 

In general our results indicate that the use of an LSP scholarship to enroll in private 
schools is associated with statistically significant—and substantively large—negative effects on 


1 The program was initially called the Student Scholarships for Educational Excellence Program but is now referred 
to as the Louisiana Scholarship Program. 

“ There are currently three private school choice programs in operation in Louisiana in addition to the Louisiana 
Scholarship Program (Lriedman Loundation for Educational Choice, 2015). The Louisiana Elementary and 
Secondary School Tuition Deduction program was implemented in 2008 to offer tax deductions to individual tax 
payers seeking to cover some of their private school expenses. The Louisiana School Choice Program for Certain 
Students with Exceptionalities initially launched in 2011 serving students with disabilities. Lastly, the Louisiana 
Tuition Donation Rebate Program, a tax-credit scholarship program, was implemented in 2012. 
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student math achievement in the first two years of the program’s statewide expansion. 
Specifically, LSP users are 34 percent of a standard deviation behind in math after attending their 
most preferred private school for two years. The magnitude of these negative estimates is 
unprecedented in the literature of random assignment evaluations of school voucher programs. In 
contrast, we observe statistically insignificant negative impact effects associated in ELA after 
two years. Finally, we present evidence indicating that the negative effects are somewhat smaller 
magnitude in Year 2 relative to Year 1, especially in math. While not conclusive, these results 
suggest the negative impacts of the program may dissipate over time. 

The remainder of this paper proceeds as follows. In the next section, we provide a brief 
background on vouchers as a policy instrument in K-12 education. Then we summarize the 
existing literature on random assignment evaluations of the participant effects of school voucher 
programs. After that, we provide a brief description of the LSP and the lottery process that 
enabled the experimental analysis. Next we describe the data and analytical strategy used to 
estimate the participant effects of the first year of the statewide expansion of the LSP. We then 
describe the results of our analyses. We conclude with a discussion of our findings. 

2. School Vouchers and K-12 Education 

School vouchers are a mechanism by which government resources are provided to families that 
enable them to attend a private school of their choosing (Wolf, 2008). Strictly speaking, a private 
school choice program is only a “voucher” program if the government funds the program directly 
out of an appropriation. Other private school choice programs are funded indirectly, through tax 
credits provided to businesses or individuals who contribute to nonprofit scholarship-granting 
organizations. Such arrangements are commonly called tax-credit scholarship programs. Since 
tax-credit scholarship programs accomplish the same general purpose as voucher programs we 
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will treat both types of private school choice programs as functionally equivalent for purposes of 
this study, although we will specify whether individual initiatives are voucher or tax-credit 
scholarship programs when discussing them. 

Although the origin of the voucher idea generally is linked to economist Milton Friedman 
(1955), political philosophers Thomas Paine (1791) and John Stuart Mill (1962 [1869]) 
supported the theoretical debate about their desirability. The theory of school vouchers is that 
government should provide funds in support of compulsory education but need not necessarily 
deliver the schooling itself. 

Whether or not students benefit from non-govemmental organizations providing their 
education is an empirical question central to the voucher debate (Doolittle & Connors, 2001). For 
example, Richard Mumane (2005) argues: 

Providing fa mi lies who lack resources with educational choices makes sense. The 

consequences of attempting to do this through a large-scale voucher...system are 

unknown. Carefully designed experiments could provide critical knowledge, (p. 

181) 

Experimental design is critical in the case of evaluating school voucher programs because 
of concerns about selection bias due to more motivated and able families self-sorting into private 
schools on their own or through access to a voucher. Fortunately, much of the research on school 
vouchers in the U.S. has taken the form of random assignment experiments. 

Prior Random Assignment Evaluations of School Voucher Programs 

Prior rigorous empirical studies of the effects of school vouchers on participants’ achievement 
have been inconsistent in their pattern of results and have yet to produce a scholarly consensus 
about the impacts of vouchers on students’ academic outcomes (Wolf, 2008; Barrow & Rouse, 
2008). The test-score results from experimental and the most rigorous quasi-experimental 
voucher studies are almost equally divided between findings of modest positive effects and 
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findings of no significant difference. To date, no rigorous evaluation of vouchers has reported 
any statistically significant negative voucher impacts. 

A total of 14 analyses have applied experimental, regression discontinuity design (RDD), 
or reliable student matching methods to data from voucher and voucher-type scholarship 
programs in Charlotte, Dayton, the District of Columbia, Florida, Milwaukee, and New York to 
determine their impacts on student achievement. Both analyses of the Charlotte data reported that 
the scholarship program produced positive and statistically significant achievement impacts 
(Greene, 2001; Cowen, 2008). The experimental evaluation of the Dayton scholarship program 
concluded that it produced achievement gains, but only for the African American subgroup of 
participants (Howell et al., 2002). A single analysis of experimental data from an early 
scholarship program in the District of Columbia concluded that achievement gains from the 
program that were evident after two years disappeared in the third and final year of the 
evaluation (Howell & Peterson, 2006). The congressionally mandated evaluation of the District 
of Columbia Opportunity Scholarship (voucher) Program, established in 2004, reported 
achievement impacts, but only in reading, that were statistically significant at a 99 percent level 
of confidence after three years (Wolf et al. 2009, p. 36) but only at a 94 percent level of 
confidence in the fourth and final year of the study (Wolf et al. 2013; Wolf et al. 2010, p. 35). An 
RDD analysis of the tax-credit scholarship program in Florida concluded that students near the 
income eligibility cutoff experienced clear achievement gains in reading, but not necessarily in 
math, if they had access to the program (Figlio, 2011). 

Two different analyses of experimental data from the early years of the Milwaukee 
voucher program reached slightly different conclusions, with one reporting that voucher students 
realized statistically significant achievement gains in both reading and math (Greene, Peterson, 
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& Du, 1999) and the other stating that the voucher achievement gains were limited to just math 
(Rouse, 1998). A more recent evaluation of the Milwaukee program concluded that a 
combination of the choice program and a high states testing policy generated test score gains but 
only in the fourth and final year of the study and only in reading (Witte et al. 2014). 

Five different analyses of data from the New York scholarship experiment also reached 
somewhat divergent conclusions. One study reported no significant achievement gains from the 
scholarship program, overall or for any subgroup of participants (Krueger & Zhu, 2004). Two 
other analyses employing alternative methods for addressing missing data found program- 
induced gains, but only for African Americans in math (Barnard, Frangakis, Hill, & Rubin, 2003; 
Jin, Barnard, & Rubin, 2010). The original experimental analysis concluded that African 
American scholarship students outperformed the control group students on a combined measure 
of math and reading scores (Mayer et al., 2002). 3 Finally, Bitler, Domina, Penner, and Hoynes 
(2013) employed quantile analysis in concluding that the program had no clear effects for 
individual subgroups along the achievement distribution. 

Four additional studies have examined the impact of vouchers on student educational 
attainment. Chingos and Peterson (2015) found no overall impact of the New York scholarship 
program on the rate of college enrollment but concluded that the African American students and 
children of parents bom in the U.S. were more likely to enroll in college and earn a bachelor’s 
degree if they had used a voucher. Wolf et al. (2013) determined that the effect of using a DC 
voucher was to increase high school graduation rates by 21 percentage points. Cowen et al. 
(2013) identified positive effects of the Milwaukee program on high school graduation, college 
enrollment, and college persistence that ranged from 4 to 7 percentage points. Warren (2011) 

3 Nevertheless, as Peterson and Howell (2004) note, Krueger and Zhu’s insignificant subgroup findings appear to be 
driven in part by the particularly unique way in which they chose to classify students as African American. 
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concluded from cross-sectional data that graduation rates were 18 percentage points higher in 
Milwaukee voucher schools compared to Milwaukee Public Schools. 

Since the pattern of results from previous experimental and RDD evaluations of voucher 
programs has ranged from neutral to positive, with no statistically significant negative impacts of 
vouchers on student achievement or attainment having been reported to date, our operating 
hypothesis at the start of our evaluation was that the LSP would have a neutral to positive impact 
on student outcomes. 

Prior Reports on the Test Score Effects of the LSP 

Several preliminary reports of the achievement effects of the LSP have been issued from the 
Louisiana Department of Education (LDE), our research team, and a second team of scholars. 

The LDE has reported annually on the achievement proficiency rates of LSP students aggregated 
to the program level and disaggregated to the school level (Louisiana Department of Education 
2013; 2014). These reports are descriptive, focus on proficiency cut points, and are used to 
evaluate and sanction, if necessary, the private schools participating in the program. Our 
evaluation, in contrast, is causal, focuses on student achievement gains relative to similar 
students who did not win the scholarship lottery, and is an evaluation of the program as a whole 
and not individual private schools. While both types of reports are helpful to policy makers and 
the public, they differ in focus and purpose. 

Our research team has presented earlier papers on the achievement effects of the LSP to 
academic audiences for their reaction and feedback. Our first presentation of one-year results 
from our evaluation was at an international conference in January of 2014 (Wolf & Mills, 2014). 
We provided additional presentations of our one-year results at national and international policy 
conferences in March of 2014 (Mills & Wolf, 2014), September of 2014 (Mills, Wolf & Greene 
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2014a; 2014b), November of 2014 (Mills, Wolf & Greene, 2014c) and January of 2015 (Mills, 
Wolf & Greene, 2015). A preliminary version of this report on the two-year effects of the LSP 
was presented at the Annual Meetings of the Association for Policy Analysis and Management in 
November of 2015 (Mills, Sude & Wolf, 2015). Thus, this particular report represents the 
culmination of a long process of research development and refinement to ensure the accuracy 
and rigor of the analysis. 

A second research team with access to the LSP lottery and student achievement data 
released a working paper reporting the one-year impacts of the program in January 2016 
(Abdulkadiroglu, Pathak & Walters, 2016). They report negative achievement effects of the LSP 
on math, reading, science and social studies that they describe as large. Their analysis only 
includes student outcomes in the first year of statewide implementation of the Louisiana voucher 
program, as students were tested eight months after switching to a participating private school. 
Our evaluation, in contrast, is longitudinal, eventually covering student achievement one, two, 
three, and four years after the initial scholarship lottery. Our experimental sample also is slightly 
larger than the sample used by the second research team. Finally, to gauge the size of the test 
score effects, the second research team compares the effects to annual student gains for the entire 
population of students in the Recovery School District of New Orleans, which includes some 
students who are not eligible for the LSP and excludes all Louisiana students outside of New 
Orleans who are eligible for the program. In this report, we instead determine the magnitude of 
the LSP achievement effects by comparing them to the distribution of test score gains of the 
randomized control group in our study, which is the ideal counterfactual to the LSP participants. 
With these distinctions in mind, readers should note that findings of Abdulkadiroglu et al. (2016) 
largely correspond with our previously presented papers on the one-year impacts of the LSP, at 
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least in math and reading. What follows is a description of our more comprehensive evaluation 
of the impacts of the LSP both one and two years after random assignment. 

3. Description of the Intervention 

The Louisiana Scholarship Program (LSP) is a statewide school voucher program available to 
moderate- to low-income students in low-performing public schools. The scholarship program is 
limited to students (1) with family income at or below 250 percent of the federal poverty line 
attending a public school that was graded C, D, or F for the prior school year according to the 
state’s school accountability system, (2) entering kindergarten, or (3) enrolled in the Recovery 
School District. In the program’s first year, 9,809 students were eligible applicants, with a 
majority of them located outside of Orleans parish. 

The LSP was created by Act 2 of the 2012 Regular Session of the Louisiana Legislature 
and Senate. Act 2 required the state board to allocate the funds for the program annually from the 
minimum foundation program. The voucher size is the lesser of the amount allocated to the local 
school system in which the student resides or the tuition charged by the participating private 
school that the student attends. Average tuition at participating private schools ranges from 
$2,966 to $8,999, with a median cost of $4,925, compared to an average total minimum 
foundation program per pupil amount of $8,500 for Louisiana public schools. 

Private schools must meet certain criteria in order to participate in the program. Those 
criteria involve (1) enrollment, (2) financial practice, (3) student mobility, and (4) health, safety 
and welfare of students. A recent survey of participating and non-participating private schools in 
Louisiana suggests that the program’s regulatory barriers have influenced schools’ choices to 
participate (Kisida, Wolf, & Rhinesmith, 2013), a possibility that we plan to explore in greater 
depth in future research. 


10 



4. Research Methodology 


Experimental Design 

When the LSP was expanded to a statewide program in 2012, the Louisiana Department of 
Education also changed the lottery process determining scholarship awards. While the original 
application process in the New Orleans pilot version of the LSP limited families to submitting 
the name of only one private school for admission, the revised application process allowed 
individuals to offer up to five private school preferences. This new lottery process is similar to 
the deferred acceptance lotteries used in New York City to assign students to schools through the 
city’s public school choice program (see Abdulkadiroglu, Pathak, & Roth, 2005). The deferred 
acceptance algorithm is designed to encourage families to reveal their true school preference 
rankings and thereby reduce the likelihood of gaming. 

While it is not the case that all eligible LSP applicants were awarded scholarships 
through a lottery process in the 2012-13 school year, we can isolate cases in which lotteries 
occurred in order to perform an experimental evaluation of the program. 

Specifically, eligible LSP applicants are allowed to submit up to five private school 
preferences and the LSP lottery algorithm attempts to place students into schools while taking 
into account several lottery priorities. Lirst, students with disabilities and “multiple birth 
siblings” 4 are manually awarded LSP scholarships if there is available space at their given school 
preference. Remaining students are grouped into one of six priority categories: 

• Priority 1 - Students who received LSP scholarships in the prior school year who are 
applying to the same school 

• Priority 2 - Siblings of Priority 1 awardees in the current round 

4 “Multiple birth siblings” are twins, triplets, etc. 
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• Priority 3 - Students who received LSP scholarships in the prior school year who are 
applying to a different school 

• Priority 4 - New applicants who attended public schools that received a “D” or “F” 
grade in Louisiana’s school accountability system at baseline 

• Priority 5 - New applicants who attended public schools that received a “C” grade in 
Louisiana’s school accountability system at baseline 

• Priority 6 - New applicants who are applying for kindergarten placements 

The first stage of the LSP award process is summarized in Figure 1. The process begins 
by attempting to place all Priority 1 category students into their first choice school. 5 The 
algorithm first groups Priority 1 students applying to the same school and grade combination and 
then checks the number of available seats for that grouping. If there are more seats than 
applicants, all students receive an LSP scholarship. If there are no seats available, no students in 
the given group receive a scholarship. Finally, if there are more applicants than seats, students 
are awarded LSP scholarships through a lottery. Once the process is complete for all Priority 1 
students, the algorithm attempts to place Priority 2 students into their first choice school. After 
cycling through all remaining priority categories, the LSP algorithm moves to the second stage 
of the allocation process by attempting to place students who have yet to receive a scholarship in 
their second choice schools. The LSP algorithm continues until all eligible applicants have either 
been awarded or not awarded an LSP scholarship. 


5 By definition, the first choice school for a priority 1 category student is the school they previously attended in the 
New Orleans pilot version of the program. 


12 




Figure 1. First stage of the Louisiana Scholarship Program award allocation process for the 
2012-13 school year. This figure illustrates the iterative process used to allocate LSP 
scholarships to students. In addition, this figure highlights the fact that only a subset of students 
was awarded LSP scholarships via lotteries. Our analysis focuses on isolating lotteries for one’s 
first choice school. 

Only a subset of eligible applicants were awarded or not awarded an LSP scholarship via 
a lottery process. Specifically, only those students in priority categories one through six whose 
school-grade combination had more applicants than available seats participated in a lottery. 
Fortunately, using data on student characteristics and school preferences, we can identify the 
subset of eligible applicants who experienced a lottery process. 6 We will focus on this subset of 
lottery participants to estimate the effects of the LSP on student achievement after two years of 
program participation because these are the only applicants for whom LSP scholarship award 


6 We identify a lottery as occurring when the percentage of students awarded an LSP scholarship falls between 0 and 
100 percent for a given school preference by grade by priority category combination. For example, if 60 percent of 
Priority 1 category students applying to third grade at school “A” as their first choice school actually received 
scholarships, we identify all students in that combination as having been subject to a lottery. 
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was randomly determined. 7 This focus on oversubscription lotteries suggests our analysis may 
be capturing the most favorable estimates of the program’s effectiveness, as higher quality 
schools are often more likely to be oversubscribed than lower quality schools (Abdulkadiroglu, 
Angrist, Dynarski, Kane, & Pathak, 2011). 

Data Description 

Most of the data for this study come from student-level datasets provided by the Louisiana 
Department of Education (LDE) in compliance with our data agreement with the state. The LDE 
provided us with their: 

• Student Information Systems (SIS) files for 2011-12 (“Baseline”) and 2012-13 (“Year 1 
Outcome”) which includes data on student enrollment and demographic background; 

• LSP eligible applicant file, which includes information on the school choice sets of all 
eligible applicants as well as the results of the 2011-12 placement lottery; 

• State assessment files for the 2011-12 and 2012-13 school years, which include data on 
each student’s participation in the annual accountability assessments and their scores. 

The Louisiana state accountability system places a strong emphasis on test-based 
accountability, with standardized assessments offered in most grades—including alternative 


7 After accounting for student testing, demographic, and school data while also limiting our analysis to students in 
binding lotteries, approximately 8 percent of the analytical sample were in Priority Category 1, 1 percent were in 
Priority Category 3, 73 percent were in Priority Category 4, and 18 percent were in Priority Category 5. Priority 
Category 2 is not represented in our analysis. 
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assessments, end-of-course exams, and exams measuring college-readiness. 8 This study uses 
student performance on the Louisiana state assessments in grades three through eight as our 
primary outcome measure of interest. 9 All students participating in the LSP are required to be 
tested by their private schools, using the state accountability assessments, for any grade in which 
the public school system also tests its students. 

In addition to individual performance outcomes, the state-provided assessment data files 
include information on student demographics as well as participation in school initiatives such as 
the free- and reduced-price lunch (FRL) program and special education program enrollments. 
Our analysis includes these baseline covariates in order to improve effect estimate precision. 10 


8 Students in Louisiana who are not classified as having a special need that qualifies them for alternative programs 
take one of two state assessments in grades three through eight. In grades four and eight, students take the Louisiana 
Educational Assessment Program (LEAP) exams, a series of criterion-referenced tests aligned with Louisiana’s state 
standards for the subjects of math, English language arts, science, and social studies. In the remaining grades, 
students take the Integrated Louisiana Educational Assessment Program (iLEAP) exams, a series of hybrid exams 
including both criterion-referenced and norm-referenced test items in the same subjects as the LEAP exams. 
Performance on both sets of exams ranges between a minimum possible score of 100 and a maximum possible score 
of 500. All exams are scaled with means of 300 and standard deviations of 50 (Louisiana Department of Education, 
2013a; 2013b). Rather than rely on these scale score values, which differ across grades by design, our analysis is 
primarily based on standardized values of individual LEAP and iLEAP performance. While the LEAP and iLEAP 
item differences introduce noise into our model, the fact that both treatment and control students in a particular 
grade take the same exam (either LEAP or iLEAP), and our inclusion of prior achievement on the right-hand side of 
the model, should reduce the likelihood of bias due to these test differences. 

9 Our initial investigation of the test databases revealed 391 eligible LSP applicants in tested grades with missing 
testing data at baseline and 516 observations in Year 1. These observations represent approximately 10 and 15 
percent of the eligible LSP applicants in relevant grade ranges for these years, respectively. Further investigation 
revealed that 82 of the missing observations took the Louisiana alternative assessments at baseline and 115 took 
them in Year 1. All records with missing baseline testing data are excluded from our analysis. In addition, a small 
number of eligible LSP applicants have duplicate records in the baseline (12 duplicate pairs) and Year 1 testing data 
(42 duplicate pairs). When possible, we have resolved duplicates by keeping records with the most complete data on 
LSP participants. For the remaining observations, we have randomly kept one record and dropped the other. These 
records represent less than 1 percent of the LSP applicants in both years. 

10 A single individual in our final analysis sample has missing data for their gender status as baseline (2011-12). We 
have updated this individual’s gender status using their reported gender in the 2012-13 assessment data. After 
making this substitution, all records in our final analysis sample have complete information on baseline covariates. 


15 



Sample Selection Process 

The student-level data provided by the LDE indicate an initial sample of 9,809 eligible LSP 
applicants in the first year of the statewide expansion of the program. Of these, 5,777 students 
received LSP scholarship placements in a specific private school and 4,038 did not receive a 
voucher-supported placement. Our analysis relies on a non-random sample of this original 
population comprised of eligible applicants with baseline testing data in grades three through six 
who did not list a special education designation on their application and who were not multiple 
birth siblings. Of the 2,541 observations meeting these criteria, we identify 1,688 individuals as 
participating in LSP scholarship lotteries. Of these, 668—or 40 percent—won LSP scholarships. 

Analytical Strategy 

We begin with a description of our primary analyses, which use the results of eligible applicants’ 
first school choice lotteries to estimate the impact of LSP scholarship usage on student 
achievement in a two-stage least squares (2SLS) framework. We then outline a series of 
subgroup analyses conducted to examine possible effect heterogeneity of the LSP. 

Local Average Treatment Effect estimation. As Bloom and Unterman (2014) note, 
because students can participate in multiple lotteries in a deferred-acceptance award process, the 
traditional intent-to-treat (ITT) estimator has limited policy relevance. 11 Instead, we estimate the 
impact of LSP scholarship usage on student achievement—also known as the Local Average 
Treatment Effect (LATE) (Angrist & Pischke, 2009, Cowen, 2008)—by using the result of one’s 
first choice school lottery as an instrumental variable to predict scholarship usage in a 2SLS 
framework. 12 The lottery is an ideal instrumental variable as the high placement take-up rate for 

11 For example, a student who loses her first lottery can still win an LSP scholarship to her second choice school via 
lottery. 

12 Prior experimental evaluations of voucher programs have varied in their focus on ITT and LATE. 
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this program ensures that it is a strong predictor of private schooling while the random nature of 
the lottery process assures that scholarship receipt is uncorrelated with unobserved factors related 
to student achievement (Murray 2006). Because the lottery is the only way a student could 
receive an LSP scholarship to attend a private school, we can be confident that the variable only 
influences student outcomes through the private schooling that it enables. 

Specifically, we use the following 2SLS model to estimate the effects of LSP scholarship 
usage on student achievement after two years: 

1. E t — Y, n jRji + 8Ti + Xfl + itj 

2. Ai = Yi a jRji + x Ei + %Y + e i 

Where i denotes student and j denotes lottery: 

• E is a variable indicating if a student used an LSP scholarship to enroll in a private 
school 13 

• Ri is a fixed effect for a student’s first choice school lottery 14 

• Ti is a variable indicating if a student received an LSP scholarship to their first choice 
school 

• Ai is student standardized math or English Language Arts achievement in Year 2 of the 
program (2013-14) 15 

• Xi is a vector of student characteristics—including achievement—collected at baseline 
( 2011 - 12 ) 


13 Prior evaluations of school voucher programs have examined enrollment effects in several ways. For example, 
Mayer et al. (2002) define enrollment as being “consistently enrolled in a private school’’, while Rouse (1998) 
defines enrollment as the number of years enrolled in an attempt to capture potential dosage effects. By defining 
enrollment as “ever attending a private school’’ our study falls in line with the Wolf et al. (2013) evaluation of the 
DC Opportunity Scholarship Program. 

14 We include a fixed effect for first school choice lottery to account for differing probabilities of success across 
lotteries (Gerber & Green, 2012). By using fixed effects, we are essentially comparing lottery winners and losers 
within the same strata to calculate unbiased estimates of the effect of being randomly offered an LSP scholarship. 
The approach is comparable to analyzing the impact of hundreds of “mini-experiments” and aggregating the results 
across them. 

15 Student achievement scores are standardized using distributional parameters of outcomes from the control group. 
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The 2SLS procedure first uses one’s treatment status to predict the number of years they 
will use a scholarship and then uses this predicted value to provide an unbiased LATE effect 
estimate (f) for the program. The proposed method instruments for LSP usage with the result of 
one’s first choice school lottery outcome (7). The 2SLS procedure will effectively treat students 
who lose their first choice lottery but go on to win an LSP to a lower school preference as 
control-group crossovers, thereby effectively excluding them from the estimated LATE (Bloom 
& Unterman, 2014). 

There are at least two types of nesting in the LSP data that could lead to biased inference 
(Angrist & Pischke, 2009). Lirst, members of both the treatment and control group are nested 
within schools in the first year of the program analysis. Second, observations can be nested 
within family units, with the potential for several children participating from the same family. 16 
This was also the case in the evaluation of the Opportunity Scholarship Program (OSP) in 
Washington, DC, in which researchers used standard errors clustered at the family level to 
account for error-covariance (Wolf et al., 2013). The results presented here do not account for 
these types of nesting due to the complex nature of multi-level clustering. Instead, we currently 
only account for nesting of observations within risk set. 17 We do not believe our results are 
strongly influenced by sibling clustering, as siblings constitute only 19 percent of our analytical 
sample. 

Subgroup analysis. In addition to estimating overall program impacts, we examine the 
extent to which LSP effects differ across different subgroups. In particular, we determine if there 
are differential impacts experienced for four subgroups: (1) males relative to females, (2) African 

16 Approximately 23 percent of individuals in our final analytical sample have siblings that also appear in the 
sample. 

17 Clustering on risk set should capture a large amount of the nesting of individuals within current school as risk set 
includes school of application. 
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American participants compared to all other program participants, (3) students in different 
baseline achievement categories, and (4) New Orleans participants compared to other 
participants. The first three comparisons are motivated by prior evaluations of school choice 
programs, which have found differential effects by gender, ethnicity, and baseline achievement 
groups. 18 The final subgroup comparison is motivated by the strong existing market for school 
choice in New Orleans in comparison with the rest of the state. In addition to having a pilot 
version of the LSP in place since 2008, New Orleans has a thriving charter school market and a 
history of public school choice (Cowen Institute, 2013). Moreover, there is evidence that the 
New Orleans reforms have increased student achievement by between 20 and 40 percent of a 
standard deviation (Harris, 2015). As such, the experiences of LSP participants in New Orleans 
may differ from other participants. 

5. Treatment-Control Contrast 

Here we examine the extent to which treatment assignment is correlated with school enrollment 
by looking at school enrollments for lottery winners and losers. We then assess the extent to 
which the lottery process resulted in covariate balance at baseline for our analysis sample. 

Scholarship Usage 

While eligible applicants were randomly assigned to receive or not receive an LSP scholarship 
with private school placement, participating families were not required to use the scholarship to 
attend their designated school. Lottery winners, for example, could choose to attend traditional 
public schools or charter schools rather than use their scholarship offer to attend the private 

18 Analyses of the New York Scholarship Program have found significant effects for African Americans, but 
insignificant effect estimates overall (Mayer et al., 2002; Barnard et al., 2003). Similarly, Wolf and colleagues 
(2013) report significant improvement in reading for female participants in the DC OSP evaluation, but no 
significant differences for males. Finally, Wolf and colleagues additionally note positive achievement effects for 
students who were already performing well at baseline. 


19 



school in which they were placed by the lottery. Lottery losers, on the other hand, could choose 
to attend a traditional public school, attend charter schools, or elect to enroll in private schools 
without a scholarship. 

Table 1 describes the patterns of enrollment for student applicants for the 2012-13 school 
year who received and did not receive LSP scholarships to their first choice schools in the first 
and second years of the program. Because our LATE analysis focuses on the results of first 
choice school lotteries, the control group includes both students who were never awarded a 
scholarship and students who received a scholarship to one of their non-first choice school 
preferences. The latter group, who account for the 61 control group members appearing in 
private schools in 2012-13 and 133 in 2013-14, are the control-group crossovers in our LATE 
analysis. 19 

The majority of lottery winners used their scholarships to attend private schools, while 
over 75 percent of students who did not receive scholarships attended public sector schools in 
both years. Sample attrition represents less than 10 percent of the analytical sample in both years; 
however it is only in Year 1 that the difference in attrition rates between treatment and control (at 
5 percentage points) is sufficiently large to merit concern (What Works Clearinghouse, 2014). 
These missing outcome observations could represent control-group crossovers attending private 
schools or students that moved out of Louisiana in the 2012-13 school year. Unfortunately, our 
reliance on the state testing data does not allow us to distinguish the causes behind these missing 
data. While our primary estimates of the effects of LSP scholarship usage on student 
achievement after two years do not account for differential attrition, we examine the sensitivity 


19 The increase in control group students attending private schools between 2012-13 and 2013-14 is explained by 
students re-applying for the program in Year 2. Our LATE analysis additionally treats these students as “non- 
compliers”. 
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of our results to differential attrition using an effect bounding exercise developed by Lee 
(2009). 20 In general, the results from our bounding analyses do not suggest that differential 
attrition has strongly influenced our primary LATE estimates. 

Table 1. 


School enrollment patterns by scholarship award 


School of Attendance 

Treatment Group 
(Received Scholarship to 
First Choice School) 

Control Group 

(Did Not Receive Scholarship 
to First Choice School) 

N 

% 

N 

% 

Year 1 (2012-13) 





Private School 

508 

77% 

54 

6% 

Public School 

123 

19% 

857 

85% 

Unknown/Missing School 

28 

4% 

93 

9% 

Year 2 (2013-14) 





Private School 

387 

59% 

126 

13% 

Public School 

215 

33% 

784 

78% 

Unknown/Missing School 

57 

9% 

94 

10% 

Total 

659 


1,004 



Notes. Sample represents all students with baseline testing data in grades three through six who did not list a special 
education exclusion on the LSP application and who were not identified as multiple birth siblings. For students in 
the treatment group attending public schools, 73 percent attended a traditional public school (TPS), 24 percent 
attended a charter school, and 2 percent attended a magnet school in Year 1. In Year 2, the corresponding 
percentages are 69 percent TPS, 27 percent charter, and 5 percent magnet. For control group students attending 
public schools in Year 1, 76 percent attended TPS, 16 percent public charters, and 8 percent attended public magnet 
schools. In Year 2, the respective percentages are 71 percent TPS, 20 percent charter, and 9 percent magnet. 

Source. Authors’ calculations. 


Baseline Equivalence 

The final step required before moving on to our empirical analysis of the participant effects of 
the statewide expansion of the LSP is to analyze the extent to which the LSP lottery process 
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“ We have two broad methods available to account for non-response bias: employing non-response weights or 
making assumptions about the nature of non-response to estimate bounds around the program’s true effect (Gerber 
& Green, 2012). Nonresponse weights effectively reweight the data to allow respondent values to account for the 
values of non-respondents (Kreuter & Valliant, 2007). Nevertheless, nonresponse weights do not account for 
potential unobservable forces that may be driving patterns of nonresponse. If, for example, those in the control 
group with higher expected outcomes both in public and private school leave the sample with higher probability, our 
LATE estimates will be positively biased. Given the likelihood that the observed control non-response reflects 
potential selection effects, we prefer instead to estimate the degree to which attrition might affect our estimates via a 
bounding exercise (Lee, 2009). Specifically, if we assume that the causes of missing data are monotonic, we can 
estimate an upper and lower bound for the LSP effect by omitting a portion of the control group from the data in 
order to balance non-response probabilities among treated and controls. 
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actually ensured independence of assignment to the treatment and control groups. While we 


cannot know the extent to which members of the treatment and control group differ on 
unobservable characteristics, we can get a good idea of the success of the lottery process by 
examining if there is baseline equivalence in observable characteristics between lottery winners 
and losers. The results of this analysis are presented in Table 2, which provides the results of t- 
tests for differences in means on key baseline covariates between members of the treatment and 
control groups included in our analysis sample, with p < .10 as the lowest threshold of statistical 
significance. 21 
Table 2. 


Baseline equivalence of treatment and control groups on covariates 



N 

Lottery 

Winners 

Lottery 

Losers 

Diff. 

s.e. 

Female 

1,663 

0.51 

0.51 

0.01 

0.03 

Race/Ethnicity 

African American 

1,663 

0.90 

0.91 

0.00 

0.02 

Hispanic 

1,663 

0.02 

0.01 

0.01 

0.01 

White 

1,663 

0.06 

0.07 

-0.01 

0.01 

Other 

1,663 

0.02 

0.01 

0.00 

0.01 

Limited English Proficiency 

1,663 

0.02 

0.01 

0.00 

0.00 

Free-or-Reduced Price Lunch 

1,663 

0.86 

0.86 

0.00 

0.01 

Number of School Preferences Listed 

1,663 

2.15 

2.35 

-0.20*** 

0.06 

Standardized Performancef 

ELA Scale Score 

1,662 

-0.35 

-0.35 

0.01 

0.05 

Math Scale Score 

1,663 

-0.39 

-0.45 

0.06 

0.05 

Science Scale Score 

1,660 

-0.47 

-0.50 

0.03 

0.05 

Social Studies Scale Score 

1,660 

-0.41 

-0.42 

0.02 

0.05 


*** - p<.01, ** - p<.05, * - p<0.10 

f Scores are standardized within grade based on the observed distributions of scale scores across Louisiana. 
Notes. The analysis sample excludes students with disabilities, multiple birth siblings, and individuals without 
baseline testing data in grades three through six All analyses include fixed effects for one’s first school choice 
lottery, “s.e.” indicates standard error of the difference, which accounts for clustering within risk sets. 

Source. Authors’ calculations 
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“ All analyses include fixed effects for one’s first school choice lottery to account for different probabilities of 
selection. 
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The results are favorable for our analysis, as nearly all of the estimated differences 
between lottery winners and losers are statistically insignificant, suggesting that we have 
adequately identified random lotteries in our analytical sample. The lone exception is that lottery 
winners provided significantly fewer school preferences on average than lottery losers. Given 
this differences, our preferred models include controls for the full set of variables examined in 
Table 2. 22 

6. Results 

This section presents the results of our primary analyses of the impacts of the statewide 
expansion of the LSP on student achievement after two years. 

Primary Estimates of the Impact of Using an LSP on Student Achievement 

The results of our primary LATE analyses are presented in Table 3. Column 1 displays 
coefficient estimates for first stage regressions using scholarship award to predict the likelihood 
of usage. 23 The results indicate that students who received an LSP scholarship to their most 
preferred school were nearly 50 percentage points more likely to attend a private school two 
years later. Columns 2 through 5 present our primary LATE estimates, with models controlling 
for an increasing number of baseline covariates as one moves from left to right in the table. 
Given the observed differences on some baseline covariates in Table 2, our preferred estimates 
of the effect of using an LSP scholarship come from the fully specified model presented in 
column 5. 


22 We present in Appendix Table At a supplemental analysis that does not require the inclusion of baseline 
achievement. The presented results do not differ substantively from our primary effect estimates (presented in Table 

3). 

23 Estimates are presented for fully specified models. In general, results from all first-stage regressions indicate that 
winning an LSP lottery to attend one’s most preferred school strongly predicts actual enrollment. 
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In general, LSP scholarship users appear to score worse than their control group 
counterparts on the state’s ELA and math exams; however the negative estimates for ELA 
achievement are not statistically significant. In contrast, LSP scholarship users score 34 percent 
of a standard deviation behind their control group counterparts in math after two years of 
enrollment in their most-preferred private school. 24 These large and statistically significant 
negative findings for math are unprecedented among experimental evaluations school vouchers 
programs in the U.S. 

Table 3. 


Estimated effects of LSP usage on student achievement after two years 


Outcome 

First Stage 

Local Average Treatment Effect 

(1) 

(2) 

(3) 

(4) 

(5) 


0 4g*** 

-0.18 

-0.18 

-0.17 

-0.18 

English Language Arts 

(0.02) 

(0.12) 

(0.12) 

(0.11) 

(0.12) 


0 4g*** 

-0 32*** 

-0.33** 

-0.33** 

-0.34** 

Mathematics 

(0.02) 

(0.11) 

(0.16) 

(0.14) 

(0.14) 

Controls 






Baseline Achievement 

X 

X 

X 

X 

X 

Demographics 

X 


X 

X 

X 

Number of Choices 

X 



X 

X 

New Orleans 

X 




X 

N 



1,525 



Risk Sets 



177 




*** - pc.Ol, ** - p<.05, * - p<0.10 

Notes. Performance measures standardized within grade based on control group score distributions. All models 
include risk set fixed effects. Standard errors (parentheses) account for clustering within risk sets. First stage F- 
statistics all exceed Staiger and Stock’s (1997) recommended threshold of 10. 

Source. Authors’ calculations. 

While the focus of this paper is on how the LSP has affected achievement over two years, 
we are also interested in the extent to which the program’s effects vary over time. Figure 2 
presents LATE estimates for ELA and math achievement separately conducted for years 1 and 2. 


24 The results presented in Table 3 do not appear to be driven by a dramatic achievement gains in the control group 
relative to the treatment group (see figures A1 and A2 in the Appendix). 
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The vertical axis represents within-grade and subject standardized achievement in 2013-14 and 


the dashed lines represent 95% confidence intervals. All models are fully specified. 
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Figure 2. Estimated Local Average Treatment Effects over time. Figure presents point estimates from fully specified 
models in Year 1 and Year 2 for both math and ELA. Dashed lines represent 95% confidence intervals for the 
performance averages. These results indicate large negative achievement effects associated with LSP scholarship 
usage after one year of program participation. By 2013-14, treatment group students appear to have been making 
small gains on the control group, with the effect on ELA achievement statistically indistinguishable from zero and 
the effect on math nearly half that observed after one year of participation. 


The findings in Figure 2 indicate that, in both years, LSP scholarship usage is associated 


with negative and statistically significant impacts on math achievement; with seemingly larger 


negative effects in Year 1. In contrast, the results for ELA indicate a statistically significant 
initial negative impact on achievement in Year 1 that dissipates to insignificance by Year 2. 25 


Thus, for both ELA and math, we find evidence suggesting the magnitude of the negative 


impacts are shrinking over time. 


25 The ELA effect is estimated with less precision in Year 2 relative to Year 1, as reflected in the larger confidence 
interval. 
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To summarize, our primary models indicate that students using LSP scholarships to 
attend their most-preferred private school have experienced negative achievement impacts, 
especially in math. While there is some evidence suggesting the magnitude of these negative 
effects may be dissipating over time, it is important to recognize that we find negative and 
statistically significant effects across all models. Relative to the existing experimental 
evaluations of voucher programs in the U.S., these negative and statistically significant findings 
are unique. At the same time, it is important to note that our evaluation is also unique among 
voucher experiments in its use of state criterion-referenced tests (CRTs) as outcome measures 
instead of norm-referenced tests (NRTs). 26 Furthermore, these results are based on a subset of 
students (approximately twenty percent of all eligible LSP applicants) who are often enrolling in 
new private schools in non-typical entry grades for schools. Thus, it is possible these effects may 
not be representative of the experiences of all LSP participants. 27 Nevertheless, the results 
presented in Table 3 indicate sizeable negative two-year achievement impacts in math associated 
with using an LSP scholarship. 28 

Subgroup Analysis 

In addition to estimating the general impacts of participation in the LSP on student achievement, 
we are interested in how various student subgroups respond to the treatment. Table 4 presents 
LATE estimates for two subgroup comparisons: females versus males and black students 


26 All of the studies discussed in section 4 used NRT exams as their primary outcome measures. In contrast, the 
legislation creating the Louisiana Scholarship Program has determined the state’s LEAP and iLEAP CRT exams to 
be the primary outcome measure of interest in the program’s evaluation. 

27 We are able to expand our sample slightly by relaxing the baseline achievement requirement in our models. While 
we observe changes in magnitude, the negative findings present in Table 3 do not change substantially when 
relaxing this requirement (see Appendix Table Al). 

28 While our analysis focuses on differences in ELA and math achievement, we have confirmed that these large 
negative effects are equally present in both science and social studies achievement. In particular, the results 
presented in Appendix Table A2 indicate that negative results for science and social studies, however only the latter 
finding is statistically significant. 
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compared to students of other race/ethnicities. 29 Columns 1 through 3 present differences across 
gender and columns 4 through 6 present models comparing black students to other students. The 
table is further divided into two panels, with Panel A presenting results for models controlling 
only for baseline achievement and risk set fixed effects and Panel B presenting results from 
fully-specified models. Joint F-statistics from first stage regressions predicting LSP usage and an 
interaction of LSP usage and a subgroup identifier are presented along with the overall results. 
Each of the reported F-statistics suggests that LSP scholarship receipt is a relevant predictor of 
usage. 

Looking first at the gender subgroup analysis, we see that, in general, the estimated math 
effects for females (column 1) and males (column 2) are not substantially different from the 
overall effect estimates presented in Table 3. This is furthermore supported by the lack of 
significant findings in column 3, the estimated differences in treatment effects between females 
and males. In ELA, the estimated effect is significant for females, but not for males. 
Nevertheless, the difference between the two groups is not statistically significant. 

The race/ethnicity subgroup analysis results presented in columns 4 through 6 indicate 
that black students and other race students did not experience significantly different effects of 
LSP scholarship usage. The effect estimates for other race students are somewhat noisy, 
however. While this is not surprising, as these students comprise a relatively small proportion of 
LSP applicants, they may have experienced substantially larger negative impacts on math 
achievement relative to black students. Nevertheless, as indicated by the insignificant difference 
estimates presented in column 6, we cannot say with any reasonable level of statistical certainty 
that the true effects differ between the two groups. 

29 

“ The results are based on the models that include terms interacting predicted LSP usage with the particular 
subgroup of interest. 
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Table 4 


Estimated effects ofLSP usage after two years, gender and race/ethnicity subgroups 



Females 

Males 

Diff. 

Black 

Students 

Other 

Students 

Diff. 


(1) 

(2) 

(3) 

(4) 

(5) 

(6) 



Panel A: Simple Model 




English Language Arts 

Mathematics 

-0.21* 

(0.12) 

-0.30** 

-0.14 

(0.19) 

-0.34** 

-0.07 

(0.20) 

0.04 

-0.18 

(0.12) 

-0.29** 

-0.13 

(0.25) 

-0.58 

-0.05 

(0.27) 

0.29 

(0.13) 

(0.17) 

(0.16) 

(0.13) 

(0.39) 

(0.35) 


Panel B: Fully Specified Model 




English Language Arts 

-0.22* 

(0.12) 

-0.14 

(0.17) 

-0.08 

(0.20) 

-0.19 

(0.13) 

-0.14 

(0.24) 

-0.05 

(0.26) 

Mathematics 

-0.33** 

-0.35** 

0.02 

-0.30** 

-0.62** 

0.32 

(0.16) 

(0.15) 

(0.16) 

(0.15) 

(0.29) 

(0.27) 

Model Summary 1 

N 


1,525 



1,525 


Risk Sets 

First Stage F Statistics 


111 



177 


LSP Use 


41.36 



54.47 


Interaction 


74.27 



57.55 



*** - pc.Ol, ** - p<.05, * - pcO.lO 

f Model summary is from fully specified math regressions. 

Notes. Panel A (Simple Model) presents results of estimations that only control for baseline achievement and risk set 
fixed effects. Panel B (Fully Specified Model) additionally controls for student demographics, number of school 
preferences offered, and geography. Performance measures standardized within grade based on control group score 
distributions. Standard errors (parentheses) account for clustering within risk sets. 

Source. Authors’ calculations. 

Finally, we examine the extent to which the estimated effects vary by baseline 
achievement and geographic location, as captured by one residing in or outside New Orleans 
when applying to the program. Table 5 presents results of regressions run separately for three 
performance subgroups (columns 1 through 3), as well as program applicants originally residing 
within and outside New Orleans (columns 4 and 5). 30 


30 

For the baseline achievement comparison, we must run separate regressions because we are unable to include both 
a control for baseline achievement and variables indicating baseline performance categories in the same model. For 
the New Orleans comparison, we must run separate regressions due to the inclusion of lottery fixed effects. Lotteries 
are defined by the school a student is applying to, which will either be located within or outside New Orleans. In 
order to identify an interaction of treatment status and New Orleans residency, a model employing lottery fixed 
effects requires variation on city of residence within treatment status within a given lottery. This is highly unlikely 
and, even when such variation exists, it is unlikely to be representative of the experience of New Orleanians and 
non-New Orleanians. We avoid this issue by estimating separate regressions. 


28 



Table 5 


Estimated effects ofLSP usage after two years, achievement and geography subgroups 



Achievement Subgroups 

Geography Subgroups 


Lower 

Middle Upper 

New Orleans 

Other 


Third 

Third Third 

Student 

Location 


(1) 

(2) (3) 

(4) 

(5) 

Panel A: Simple Model 




English Language Arts 

0.25 

(0.33) 

-0.16 -0.42*** 

(0.16) (0.16) 

-0.20 

(0.64) 

-0.16 

(0.10) 

Mathematics 

0.24 

(0.36) 

-0.38 -0.44** 
(0.28) (0.18) 

-0.82 

(0.96) 

-0.27** 

(0.11) 

Panel B: Fully Specified Model 



English Language Arts 

0.05 

(0.31) 

-0.12 -0.41*** 

(0.17) (0.14) 

-0.32 

(0.62) 

-0.15 

(0.10) 

Mathematics 

0.32 

(0.40) 

-0.40* -0.49** 
(0.21) (0.19) 

-0.85 

(0.73) 

-0.26** 

(0.11) 

Model summary! 

N 

509-512 

508-513 503-505 

254 

1,271 

Risk Sets 

First Stage 

146 

145 142 

67 

142 

LSP Use 

0.57 

0.48 0.40 

0.34 

0.52 

(0.05) 

(0.06) (0.05) 

(0.08) 

(0.02) 

Joint F-statistic 

224.49 

92.15 150.89 

29.26 

755.00 


*** - pc.Ol, ** - p<.05, * - p<0.10 

f Model summary is from fully specified math regressions. 

Notes. Panel A (Simple Model) presents results of estimations that only control for baseline achievement and risk set 
fixed effects. Panel B (Fully Specified Model) additionally controls for student demographics, number of school 
preferences offered, and geography. Performance measures standardized within grade based on control group score 
distributions. Standard errors (parentheses) account for clustering within risk sets. 

Source. Authors’ calculations. 

While nearly all point estimates suggest a negative effect, the estimates themselves are 
often quite noisy. Among performance categories, it appears that only students performing in the 
upper third of the achievement distribution at baseline have statistically significant negative 
impacts. Students initially performing in the upper third of the achievement distribution ended up 
40 percent of a standard deviation behind their control group counterparts in ELA and math after 
two years. In contrast, students initially scoring in the lower end of the performance distribution 
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do not appear to have experienced significant negative achievement impacts associated with LSP 
scholarship usage. 

In addition, the LATE estimates are noticeably more negative for students in New 
Orleans (column 4); however these estimates are particularly noisy given the small number of 
students contributing to the analysis. The estimates for non-New Orleans students are slightly 
smaller than the main effects presented in Table 3, further providing some evidence that the 
relatively more negative, yet noisy, effects observed in the New Orleans subsample of students is 
pulling down the overall effect. 

Robustness Checks 

In general, our analyses indicate that participation in the statewide expansion of the LSP 
negatively impacted student achievement on Louisiana’s state assessments after two years. These 
negative findings are unique among random assignment evaluations of school voucher programs, 
all of which have found insignificant or positive outcomes. In this section, we present two 
sensitivity analyses designed to test the robustness of our findings. 

Sensitivity of results to differential attrition. Our first robustness check examines the 
extent to which our estimated effects are sensitive to the different rates of attrition observed 
between treatment and control group members in our sample in the 2012-13 school year. 
Specifically, we find that 93—or 9 percent—of students who did not win an LSP to their first 
choice school do not appear in the state’s assessment data in 2012-13; whereas only 4 percent of 
LSP winners are missing. While this difference is not cause for great concern (What Works 
Clearinghouse, 2014), it is important to consider if differential attrition is driving our primary 
findings. This subsection focuses exclusively on differential attrition in 2012-13, as the sample 
attrition rates are roughly equal in the 2013-14 school year. 
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If we can assume the observed differences in attrition are due to random factors, our 


LATE estimates are generally less precise but are not biased by differential attrition (Gerber & 
Green, 2012). On the other hand, if the observed differences are due to systemic, yet 
unobservable, sample selection effects, our primary estimates of the effect of using an LSP 
scholarship on student achievement are biased (Gerber & Green, 2012; Lee, 2009). If, for 
example, those in the control group with higher expected outcomes both in public and private 
school leave the sample with higher probability, our LATE estimates will be positively biased. 

Here, we examine the extent to which differential attrition may be biasing our results by 
using a bounding strategy. In particular, we use a technique developed by Lee (2009)—hereafter 
referred to as “Lee Bounds”—that involves removing a subset of applicants from the treatment 
group in an attempt to parse out marginal individuals who have selected into the sample only 
because they received an LSP scholarship. 31 In particular, Lee shows that if one can assume that 
problematic attrition is only present in either the treatment or control group, then one can 
effectively bound the average treatment effect for individuals whose treatment status does not 
influence their sample selection likelihood by trimming away from that group a percentage of 
applicants equal to the attrition difference from the bottom and top performers. These trimming 
procedures produce upper and lower bounds of the effect, respectively. 32 


31 Lee’s (2009) bounding method is built on two assumptions: that the assignment mechanism is random and that 
sample selection is a monotonic function of treatment status. The first assumption is easily satisfied by the LSP 
lottery process. The second assumption essentially requires that there are no LSP applicants who were assigned an 
LSP scholarship but decided to forgo their scholarship and instead enroll in a private school at their own expense. 
While we cannot validate this assumption empirically, it seems highly unlikely that such “defiers” exist in our 
data—especially given the program’s income threshold. 

32 One of the primary benefits of Lee’s bounding method is that it does not require strong assumptions on the 
selection mechanism producing the attrition problems beyond the assumption that the effect is only present in either 
the treatment or control groups. For example, one need not assume that control group attriters are either more- or 
less-academically able than students who actively choose to remain in the sample. Nevertheless, the simplicity of 
Lee’s method comes at a cost: Lee bounds can be quite large—especially in the presence of large differences in 
nonresponse rates 
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Table 6 presents both the original LATE estimates produced in Table 3-included as a 
reference—as well as results from the two bounding exercises described. Columns 2 through 4 
present models controlling only for baseline achievement and risk set while columns 5 through 7 
present models that additionally include controls for demographics and residence. 

As expected, the Lee bounds presented in Table 6 are quite large, with differences 
between lower and upper bounds of over 20 percent of a standard deviation in achievement. 
Despite the magnitude of these gaps, the results for math are consistent with LSP scholarship 
usage having a negative effect on achievement. In contrast, the results for ELA suggest that—in 
a best case scenario—LSP scholarship usage may have had an insignificant effect on student 
achievement after one year. Given the magnitude of the estimated effect, along with the 
knowledge that these estimates are based on removing the lowest performers from the treatment 
group, we are hesitant to conclude that the overall effect of the LSP on ELA achievement was 
null in the first year of the statewide expansion. Nevertheless, if this assumption is met, the 
results presented in Table 6 suggest the possibility that the two groups did not differ in ELA 
achievement after one year. 

In general, the results presented in this section do not suggest that differential attrition has 
strongly biased the primary results presented in Table 3. Specifically, unless we make fairly 
restrictive assumptions, LSP scholarship usage continues to be associated with negative impacts 
in both math and ELA achievement. While our upper bound estimate of the LSP effect on ELA 
achievement using Lee’s bounds is indeed statistically insignificant, we caution the reader 
against using this extreme estimate to serve as the program’s effect on student achievement after 
one year. 
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Table 6 . 


Accounting for differential attrition in Year 1 





Simple Model 


Lully Specified Model 


N 

Lower 

Primary LATE Bound 

Upper 

Bound 

Primary LATE 

Lower 

Bound 

Upper 

Bound 

( 1 ) 

( 2 ) 

(3) 

(4) 

(5) 

( 6 ) 

(7) 




Panel A: ELA Achievement 




Primary LATE 

1,537 

-0.18*** 


- 0 . 20 *** 





(0.06) 



(0.08) 



Lee Bounds 

1,503 - 1,509 


-0 27*** 

-0.07 


-0.28*** 

-0.09* 




(0.06) 

(0.06) 


(0.07) 

(0.05) 




Panel B: Math Achievement 




Primary LATE 

1,538 

-0.63*** 


-0.64*** 





(0.07) 



(0.09) 



Lee Bounds 

1,504- 1,510 


-0 70*** 

-0 49 *** 


-0 71*** 

-0.50*** 




(0.07) 

(0.08) 


(0.08) 

(0.08) 


*** - pc.Ol, ** - p<.05, * - p<0.10 

Notes. “Simple Model” refers to estimations that only control for baseline achievement and risk set fixed effects. “Fully Specified Model” refers to estimations 
that additionally control for student demographics, number of school preferences offered, and geography. Performance measures standardized within grade based 
on control group score distributions. Standard errors (parentheses) account for clustering within risk sets. 

Source. Authors’ calculation 
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Curricular advantage. It is important to recognize that our analyses are based on 
achievement on the Louisiana state assessments, rather than nationally representative exams. 
These results may simply reflect the fact that public schools are operating with curricula that are 
already aligned with the state assessments, while private schools have yet to align their curricula. 

While we cannot provide an exhaustive examination of the teaching methods of the 
private schools in our sample, our testing data allow us to partially examine this question. We 
test for a curricular advantage by making use of the fact that some of the Louisiana state 
assessments include both criterion-referenced and norm-referenced exam questions. In particular, 
while the Louisiana assessments in grades four and eight only include criterion-referenced items, 
the iLEAP assessments offered in grades three, five, six, and seven include both criterion- and 
norm-referenced exam questions. If public school students experience a disproportionate 
curricular advantage, one would expect smaller negative LATE impact estimates on the i LE AP 
exams than on the LEAP exams. 

Table 7 presents results from models examining the extent to which LSP usage effects 
differ across test type. Unlike in our earlier analyses, the results presented in Table 7 focus on the 
first year of program participation because we believe the shock of a curricular disadvantage to 
be greatest for private schools participating in the first year of the program. 

For both ELA and math, we find that students taking the LEAP exam do appear to 
perform worse than iLEAP takers; however the estimated differences are statistically 
insignificant all cases. While not definitive, this pattern of results suggests that the substantial 
negative LATE impact estimates could be partially driven by the stronger alignment of the public 
school curricula to the state assessments. At the same time, it is important to note that LSP 
scholarship users still performed quite poorly on the hybrid iLEAP exams. Thus, while these 
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findings may provide some insight into the substantial magnitude of our estimated impacts, they 
nevertheless support the general finding of a negative overall effect of the program after one 
year. 

Table 7 


Estimated effects ofLSP usage, iLEAP vs. LEAP test takers after one year 



iLEAP Takers LEAP Takers 

Diff. 


(1) 

(2) 

(3) 


Panel A: Simple Model 



English Language Arts 

-0.14* 

(0.09) 

-0.24 

(0.18) 

0.10 

(0.21) 

Mathematics 

-0.58*** 

(0.07) 

-0 77*** 

(0.21) 

0.19 

(0.25) 


Panel B: Fully Specified Model 



English Language Arts 

-0.15* 

(0.08) 

-0.29 

(0.20) 

0.13 

(0.22) 

Mathematics 

-0.59*** 

(0.09) 

-0 79*** 

(0.21) 

0.20 

(0.22) 

Model summary 1 

N 


1,538 


Risk Sets 

First Stage F Statistics 


176 


LSP Use 


38.54 


Interaction 


121.42 



*** - pc.Ol, ** - p<.05, * - p<0.10 

f Model summary is from fully specified math regressions. 

Notes. Panel A (Simple Model) presents results of estimations that only control for baseline achievement and risk set 
fixed effects. Panel B (Fully Specified Model) additionally controls for student demographics, number of school 
preferences offered, and geography. Performance measures standardized within grade based on control group score 
distributions. Standard errors (parentheses) account for clustering within risk sets. 

Source. Authors’ calculations. 

7. Conclusion 

This paper presents an estimation of the effects of the statewide expansion of the Louisiana 
Scholarship Program (LSP)—one of the newest and largest school voucher programs in the 
U.S—on student achievement after two years. This study contributes to the existing literature on 
the participant effects of publicly funded voucher programs for two reasons. First, it uses a 
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highly rigorous experimental design to estimate treatment effects while avoiding self-selection 
bias concerns. Second, it is among the first evaluations of a statewide school voucher program. 
These contributions will add to the existing knowledge on the effects of private school choice 
programs. 

The results presented in this paper indicate significant and substantial negative 
achievement impacts associated with using an LSP scholarship. In general, we find that LSP 
scholarship usage is associated with declines of 20 percent of a standard deviation in ELA 
achievement and 35 percent of a standard deviation in math, however only the latter finding is 
statistically significant. These findings are the first of their kind among random assignment 
evaluations of school voucher programs and are robust to several alternative specifications. 33 

At the same time, it is important to keep in mind that our analyses are based on a small 
subsample of LSP participants with performance data on the Louisiana state assessments. 
Specifically, our analysis sample represents approximately 20 percent of the 2012 cohort of 
eligible applicants. Thus, in a real sense, this paper is not an evaluation of the entire program, but 
an evaluation of the experiences of students in grades three through seven at baseline, who 
participated in actual lotteries, with testing outcomes in Year 2. The educational impact of the 
LSP on the many thousands of program participants who do not satisfy those criteria remains, at 
this point, unknown. Readers are encouraged not to draw firm conclusions from this initial 
analysis due to the severe threats to external validity posed by those limitations of the sample. 


33 The estimates are not sensitive to the inclusion of baseline test data (Appendix Table Al). We find similar 
relationships when examining science (insignificant negative results) and social studies (significant negative 
estimates) (Appendix Table A2). Results do not appear to depend on our definition of what constitutes a lottery 
(Appendix Table A3). Finally, we find similar results in a quasi-experimental analysis examining the relationship 
between private school attendance and student achievement (Appendix Table A4). 
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At this point we can only speculate as to why our results differ so dramatically from the 
voucher experiments conducted previously. Specifically, we can offer four explanations that 
coincide with the observed findings. 

The first explanation deals with the LSP’s scale. As mentioned, this is the first 
experimental evaluation of a school voucher program implemented state-wide. In contrast, the 
nonsignificant and modestly positive impact estimates associated with earlier programs are 
drawn from programs largely serving small samples of students in urban school districts. Among 
these studies, the evaluation of the DC Opportunity Scholarship Program (Wolf, et al., 2013) is 
the most comparable experimental evaluation of a voucher program in scope and scale. While 
Wolf et al. (2013) found large positive attainment effects associated with the DC OSP, they 
found modest achievement effects, only observed in reading and largely concentrated among 
relatively advantaged students. Moreover, recent non-experimental yet rigorous analysis of a 
statewide voucher program in Indiana also reports significant negative achievement effects in the 
short-run that decrease in size over time (Waddington & Berends, 2015). Such findings suggest 
that the LSP’s scale of implementation may have played a role in the significant negative effect 
estimates presented here. 

A second explanation lies in the relatively short implementation time frame. The 
statewide expansion of the LSP was passed during the end of the 2012 Louisiana State 
Legislative Session in June and participating schools did not receive information on their 
incoming students until the August of 2012, giving the schools little time to prepare for their new 
students. Moreover, participating private schools had only six months to prepare the new 
students for the Louisiana state assessments, tests aligned to the state’s standards and to which 
the private schools had never before been subject. While we do not find strong evidence that 
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private schools faced a curricular disadvantage by being subject to the state’s assessments, the 
declining magnitude of the negative effects observed in ELA and math do coincide with an 
adjustment period for participating private schools. Moving forward, it will be important to 
determine if the negative initial impacts continue to dissipate over time. 

A third explanation deals with the pool of students eligible for the LSP. While most of 
the earlier voucher programs examined by experimental evaluations focused on serving 
disadvantaged sub-populations of students, none of the students in those evaluations were 
required to have attended poorly performing public schools as is the case for the LSP. This 
additional academic requirement could explain the substantial drop off in performance if 
participating private schools were not adequately prepared to serve the needs of students who 
were both financially and academically in great need. While the doubly-disadvantaged nature of 
LSP participants is a possible explanation for the observed negative effects, it is not a 
justification for them. The LSP eligibility requirements are an important design feature of the 
program and are reflective of program goals. That participating private schools struggled to meet 
the needs of such students in the first year of the state’s implementation suggests the program did 
not meet its goals in that first year. The fact that the large achievement gap between the LSP and 
control group students after Year 1 had declined somewhat in Year 2, especially in math, 
suggests that participating schools successfully adjusted to meet the significant needs of their 
new students. As with the earlier explanations, more time is needed to determine the extent to 
which the earlier observed negative effects persist in the long run. 

Linally, it could be the case that a higher-quality set of private schools participated in 
earlier voucher and scholarship programs in Washington, DC; New York City; Dayton, Ohio; 
Milwaukee, Wisconsin; and Charlotte, North Carolina; in which more positive voucher 
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experimental impacts were reported. Our initial descriptive analysis of data about participating 
and non-participating Louisiana private schools generally supports that hypothesis (Mills, Sude 
& Wolf, 2015). Less than one-third of the private schools in Louisiana chose to participate in the 
LSP in its first year, possibly because of the extensive regulations placed on the program by 
government authorities (Kisida, Wolf, & Rhinesmith, 2015) combined with the relatively modest 
voucher value relative to private school tuition (Mills, Sude & Wolf, 2015). Although it is only 
speculation at this point, the Louisiana Scholarship Program regulatory requirements may have 
played a role in preventing the private school choice program from attracting the kinds of private 
schools that would deliver better outcomes to its participants. 

Nevertheless, while certain aspects of the findings presented here align with each of these 
explanations, it is important to recognize that they are, at this point, simply speculations. A 
limitation of this research is that our design cannot test these hypotheses conclusively. Instead, 
the purpose of this work is to provide the most rigorous assessment of the effect of the program 
on student achievement. In this regard, it is clear the LSP has negatively affected the 
achievement of the subset of eligible participating LSP students examined here. Most likely, 
each of the four explanations offered here played a role in the negative findings we observe. 
Additional research, including examination of long-run and non-cognitive effects and qualitative 
analyses aimed at understanding the program’s implementation, is needed to help shed light on 
the negative achievement findings presented here. 
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Appendix 



2011-12 2012-13 2013-14 


Did not win first choice lottery M Won first choice lottery 

Figure Al. Comparison of treatment and control group average ELA performance over time. 
Achievement has been standardized by grade and year to the Louisiana state test taking 
distribution. Dotted lines represent 95% confidence intervals for the performance averages. 

These results indicate that control group students did experience a mild improvement relative to 
the state over time; however treatment group students experienced a large decline in performance 
between 2011-12 and 2012-13. By 2013-14, treatment group students appear to have been 
making small gains on the control group. 
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Did not win first choice lottery M Won first choice lottery 


Figure A2. Comparison of treatment and control group average math performance over time. 
Achievement has been standardized by grade and year to the Louisiana state test taking 
distribution. Dotted lines represent 95% confidence intervals for the performance averages. 

These results indicate that control group students did experience a mild improvement relative to 
the state over time; however treatment group students experienced a large decline in performance 
between 2011-12 and 2012-13. By 2013-14, treatment group students appear to have made small 
gains on the control group. 
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Table A1 


Removing the baseline achievement requirement 



Year 1 

Year 2 


w/ Baseline 
Achievement 

No Baseline 
Achievement 

w/ Baseline 
Achievement 

No Baseline 
Achievement 

Outcome 

(1) 

(2) 

(3) 

(4) 

English Language Arts 

Mathematics 

-0.20*** 

(0.08) 

-0.64*** 

(0.09) 

-0 23*** 

(0.08) 

-0.54*** 

(0.08) 

-0.18 

(0.12) 

-0.34** 

(0.14) 

-0.21* 

(0.11) 

-0.53*** 

(0.15) 

N 

1,537 

2,363 

1,525 

2,239 

Risk Sets 

176 

258 

177 

256 


*** - pc.Ol, ** - p<.05, * - p<0.10 

Notes. All models are fully specified. Performance measures standardized within grade based on control group score 
distributions. All models include risk set fixed effects. Standard errors (parentheses) account for clustering within 
risk sets. 

Source. Authors’ calculations. 
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Table A2. 


Estimated effects ofLSP usage on student achievement after two years, multiple outcomes 


Outcome 


LATE Estimates 


(1) 

(2) 

(3) 

(4) 

English Language Arts 

-0.18 

-0.18 

-0.17 

-0.18 


(0.12) 

(0.12) 

(0.11) 

(0.12) 

Mathematics 

-0 32*** 

-0.33** 

-0.33** 

-0.34** 


(0.11) 

(0.16) 

(0.14) 

(0.14) 

Science 

-0.16 

-0.15 

-0.15 

-0.16 


(0.13) 

(0.12) 

(0.10) 

(0.11) 

Social Studies 

-0 40*** 

-0 40*** 

-0.39** 

-0 40*** 


(0.13) 

(0.13) 

(0.15) 

(0.12) 

Controls 





Baseline Achievement 

X 

X 

X 

X 

Demographics 


X 

X 

X 

Number of Choices 



X 

X 

New Orleans 




X 

N 


1,525 


Risk Sets 


177 



*** - pc.Ol, ** - p<.05, * - p<0.10 

Notes. Performance measures standardized within grade based on control group score distributions. All models 
include risk set fixed effects. Standard errors (parentheses) account for clustering within risk sets. First stage F- 
statistics all exceed Staiger and Stock’s (1997) recommended threshold of 10. 

Source. Authors’ calculations. 
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Table A3. 


Estimated effects ofLSP usage across different lottery specifications 




Risk Set Win Percentage 



0-100 

5-95 

10-90 

20-80 


( 1 ) 

( 2 ) 

(3) 

(4) 


Panel A: Simple Model 




English Language Arts 

-0.18 

( 0 . 12 ) 

-0.18 

( 0 . 11 ) 

-0.16 

( 0 . 11 ) 

-0.19* 

( 0 . 11 ) 

Mathematics 

-0.32** 

(0.15) 

-0.32** 

(0.16) 

-0.32** 

(0.14) 

-0.25* 

(0.13) 


Panel B: Fully Specified Model 



English Language Arts 

-0.18 

( 0 . 12 ) 

-0.18 

( 0 . 12 ) 

-0.18* 

(0.09) 

- 0 . 22 * 

( 0 . 12 ) 

Mathematics 

-0 34*** 
( 0 . 12 ) 

-0 34 *** 
( 0 . 12 ) 

-0.35*** 

(0.13) 

-0.30* 

(0.17) 

Model summary* 

N 

1,525 

1,512 

1,329 

919 

Risk Sets 

First Stage 

177 

176 

169 

136 

LSP Use 

0.49 

(0.03) 

0.49 

(0.03) 

0.45 

( 0 . 02 ) 

0.50 

(0.03) 

Joint F-statistic 

478.74 

381.22 

534.53 

309.91 

*** - pc.Ol, ** - p<.05, * - p<0.10 

f Model summary is from fully specified math regressions. 


Notes. Table examines sensitivity of results to alternative definitions of lotteries that use more restrictive range for 
risk set win percentages. Panel A (Simple Model) presents results of estimations that only control for baseline 
achievement and risk set fixed effects. Panel B (Fully Specified Model) additionally controls for student 
demographics, number of school preferences offered, and geography. Performance measures standardized within 
grade based on control group score distributions. Standard errors (parentheses) account for clustering within risk 
sets. 

Source. Authors’ calculations. 
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Table A4. 


Quasi-experimental analysis of private school attendance 

_ Year 1 _ 

Simple Model Fully Specified 

_ ( 1 ) ( 2 ) 

0 19*** _q i 9 *** 

(0.04) (0.03) 

q ^2*** -0 52*** 

(0.04) (0.05) 


English Language Arts 
Mathematics 


_ Year 2 _ 

Simple Model Fully Specified 


( 3 ) 

-0.18*** 
(0.04) 
-0 44*** 
(0.05) 


( 4 ) 

-0.19*** 
(0.04) 
-0 44*** 
(0.04) 


Model summary 

N 2,315-2,322 2,249-2,255 

R-squared _ 0.60-0.61 _ 0.49 - 0.52 _ 

*** - pc.Ol, ** - p<.05, * - p<0.10 

Notes. Results are based on OLS models examining the relationship between private school attendance and student 
achievement. Performance measures standardized within grade based on the performance of all test takers in the 
state of Louisiana in the given year. This differs from our primary analysis, which standardizes performance relative 
to the control group test distributions. “Simple Model" refers to estimations that only control for baseline 
achievement and risk set fixed effects. “Fully Specified Model” refers to estimations that additionally control for 
student demographics, number of school preferences offered, and geography. All models include risk set fixed 
effects. Standard errors (parentheses) account for clustering within risk sets. 

Source. Authors’ calculations. 
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